RESUMO
Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.
RESUMO
BACKGROUND: Video teleconferencing (VTC) as a substitute for in-person health care or as an adjunct to usual care has increased in recent years. PURPOSE: To assess the benefits and harms of VTC visits for disease prevention, diagnosis, and treatment and to develop an evidence map describing gaps in the evidence. DATA SOURCES: Systematically searched PubMed, EMBASE, Web of Science, and the Cochrane Library from 1 January 2013 to 3 March 2021. STUDY SELECTION: Two investigators independently screened the literature and identified 38 randomized controlled trials (RCTs) meeting inclusion criteria. DATA EXTRACTION: Data abstraction by a single investigator was confirmed by a second investigator; 2 investigators independently rated risk of bias. DATA SYNTHESIS: Results from 20 RCTs rated low risk of bias or some concerns of bias show that the use of VTC for the treatment and management of specific diseases produces largely similar outcomes when used to replace or augment usual care. Nine of 12 studies where VTC was intended to replace usual care and 5 of 8 studies where VTC was intended to augment usual care found similar effects between the intervention and control groups. The remaining 6 included studies (3 intended to replace usual care and 3 intended to augment usual care) found 1 or more primary outcomes that favored the VTC group over the usual care group. Studies comparing VTC with usual care that did not involve in-person care were more likely to favor the VTC group. No studies evaluated the use of VTC for diagnosis or prevention of disease. Studies that reported harms found no differences between the intervention and control groups; however, many studies did not report harms. No studies evaluated the effect of VTC on health equity or disparities. LIMITATIONS: Studies that focused on mental health, substance use disorders, maternal care, and weight management were excluded. Included studies were limited to RCTs with sample sizes of 50 patients or greater. Component analyses were not conducted in the studies. CONCLUSION: Replacing or augmenting aspects of usual care with VTC generally results in similar clinical effectiveness, health care use, patient satisfaction, and quality of life as usual care for areas studied. However, included trials were limited to a handful of disease categories, with patients seeking care for a limited set of purposes. PRIMARY FUNDING SOURCE: Patient-Centered Outcomes Research Institute.
